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1  SUMMARY 


The  goal  of  this  task  is  to  design  a  novel  way  of  learning  and  real-world  problem  solving  by 
determining  the  optimal  human-computer  symbiotic  learning  and  problem  solving  frame- work. 
There  are  no  general  methods  for  discovering  optimal  learning  pathways  to  bring  novices  to 
experts  in  an  area,  or  how  to  effectively  combine  humans  and  computers  in  a  general  framework. 
The  effort  specifically  focused  on  problem  of  critical  importance  to  the  military,  visual  cognitive 
skills  required  for  Improvised  Explosive  Device  (IED)  discovery. 


The  effort  used  a  game-based  framework  to  collect  large  quantities  of  data  on  the  possible 
improvements  in  the  performance  of  regular  people  on  the  low-level  visual  cognitive  tasks  of  high 
importance  for  IED  discovery  as  identified  by  Joint  Improvised  Explosive  Device  Defeat 
Organization  ( JIEDDO).  IED  discovery  game  provides  a  venue  to  assist  in  teaching/training 
individuals  for  real  world  crisis  situations.  The  effort  focused  on  developing  a  set  of  skills  necessary  to 
enhance  IED-relevant  visual  skills:  anomaly  detection,  visual  perception,  and  useful  field  of  view. 
The  IED  discovery  game  served  both  as  a  teaching  tool  and  as  a  way  to  understand  how  to 
incorporate  those  skills  into  more  elaborate  IED  training  tools. 


The  findings  showed  that  even  with  voluntary  play  people  improve  on  psychological  tests.  On  fully 
controlled  tests  with  pre  and  post  psychological  tests  we  found  marked  improvement  on  just  10  hours 
of  play.  Specifically,  players  improved  on  visual  search  in  tenns  of  time  required  for  the  same 
accuracy  of  results.  The  largest  effect  was  found  on  useful  field  of  view  cognitive  tasks  where 
people  improved  their  performance  up  to  30%  on  controlled  tests  after  10  hours  of  play.  These 
findings  point  out  that  a  targeted  adaptive  game  is  a  powerful  training  tool  capable  of  significantly 
improving  low-level  visual  cognition  tasks  with  only  10  hours  of  play. 
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2  INTRODUCTION 


The  approach  relies  on  the  ability  to  gather  discriminative,  nuanced,  massive  data  set  on  how 
people  approach  learning  about  and  solving  the  problem.  Therefore,  the  effort  used  a  game  based 
framework,  as  games  are  an  endeavor  that  people  naturally  gravitate  towards  and  spend  time  on. 
The  game  framework  puts  game  based  problem  solving  as  a  principal  mechanism  for  learning  and 
problem  solving.  The  game  can  continually  adapt  to  data  gathered  in  an  interactive  process  of 
design  and  refinement. 


In  contrast  to  standard  curriculum-based  approach,  we  dynamically  optimize  for  optimal  learning 
and  problem  solving  pathways  within  a  rapidly  adapting  video  game  framework.  The  game-based 
approach  will  enable  engagement,  return,  and  retention  of  players.  Simultaneously,  it  provides 
access  to  massive  amounts  of  data  on  learning  and  problem  solving  that  can  be  used  to  discover 
optimal  pathways. 
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3  METHODS,  ASSUMPTIONS,  AND  PROCEDURES 


A  team  of  developers,  designers,  and  artists  has  developed  and  deployed  a  prototype  of  the  IED 
discovery  game.  The  effort  identified  the  target  skills  to  train  with  the  game  and  methods  for 
evaluating  and  initial  game  mechanics  to  train  skills.  The  design  for  the  prototype  includes 
maximum  flexibility  for  iterative  design  and  adaption  as  well  as  experimentation  to  discover 
optimal  pathways. 


Figure  1:  Vampire  Vision  Screenshots 

The  effort  specifically  focused  on  the  problem  areas  of  critical  importance  to  the  military,  visual 
acuity  skills  relevant  for  IED  discovery  and  detection  tasks.  The  objective  was  to  develop  and 
release  a  game  to  perform  an  adaptive  mass-scale  randomized  trial  that  determines  the  best  method 
to  train  a  specific  skill  set.  Meeting  the  objective  was  the  first  step  to  developing  a  more  general 
framework  for  skill  training. 

The  game  is  called  Vampire  Vision.  A  screenshot  is  shown  in  Fig.  1.  In  the  game,  the  player  is 
meant  to  find  vampires  hiding  amongst  humans.  The  properties  that  identify  vampires  constantly 
change  so  that  the  player  cannot  learn  to  profile  and  must  generalize  their  search  strategy.  There  are 
different  types  of  levels,  each  with  a  unique  play  style  that  corresponds  to  a  visual  perception  skill 
we  want  to  train.  Visual  perception  tests  are  embedded  in  the  game  at  various  intervals  so  that  we 
can  measure  the  effect  of  the  game  on  players’  skills. 
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4  RESULTS  AND  DISCUSSION 


This  effort  released  the  game  onto  a  number  of  distribution  portal  sites,  such  as  newgrounds.com. 
The  game  has  been  picked  up  by  more  distribution  portals,  with  hundreds  of  new  players  each  day. 
The  game  has  been  played  over  600,000  times. 

Over  time,  this  effort  was  able  to  gather  and  analyze  a  variety  of  data  from  those  players.  This  data 
was  used  to  iteratively  improve  the  design  of  the  game. 

The  effort  was  able  to  rapidly  deploy  multivariate  experiments  using  the  game,  focusing  on  such 
aspects  as  game  mechanics  for  player  retention  (such  as  the  effects  of  including  achievements), 
properties  of  embedded  tests  (such  as  the  tradeoff  involved  in  the  length  of  embedded  tests),  and 
optimal  level  progressions  (such  as  the  effects  of  the  difficulty  increases  as  players  progress  through 
the  game).  The  effort  included  more,  improved,  and  better  parameters  to  visual  tests  in  the  game. 
The  effort  integrated  a  reinforcement  learning  framework  to  discover  pathways  for  optimizing  these 
metrics  within  the  game,  based  on  the  large  amounts  of  data  we  can  gather. 

The  game  has  been  greatly  improved  and  polished  from  is  initial  release.  The  game  architecture  is 
flexible  and  allows  for  a  variety  of  rapid  experiments  and  data  gathering  and  analysis  to  discover 
optimal  progressions.  More  game-based  mechanics,  such  as  an  achievement  system,  have  been 
added.  The  game  has  been  refined  for  a  more  intuitive,  usable  interface  to  attract  and  retain  players. 

Features  of  the  “meta-game”  have  been  implemented,  such  as  level  selection,  that  will  be  more 
focused  toward  social  gaming  sites.  The  experimentation  framework  used  in  the  game  has  been 
generalized  so  that  it  can  be  applied  to  more  games. 

A  controlled  lab  study  was  carried  out  by  members  of  Geoffrey  M.  Boynton’s  lab  who  were  not 
funded  by  this  grant.  Through  lab  experiments,  they  found  the  useful  field  of  view  (UFOV)  test  to 
be  the  most  promising  candidate  for  improvement,  as  well  as  the  visual  search  (VS)  test.  Results  of 
the  UFOV  test  can  be  seen  in  Fig.  2  and  for  the  VS  test  in  Fig.  3.  It  was  found  that  players  of  the 
game  improved  their  ability  in  UFOV  tests  and  their  speed  in  VS  tests  when  compared  to  a  control 
game  by  approximately  30%. 

Developing  and  testing  improved  UFOV  and  VS  tests  for  deployment  remains  an  open  area.  These 
improved  tests  could  then  be  embedded  in  the  online  version  of  the  game  for  improved  evaluation 
of  player  skills. 
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Figure  2:  Useful  Field  of  View  Comparison 


5  CONCLUSIONS 

This  effort  developed  a  game,  Vampire  Vision,  that  focused  on  training  visual  perception  skills 
through  gameplay  and  evaluating  them  with  embedded  tests.  The  game  was  widely  deployed  on 
the  internet  through  online  gaming  portals  and  found  an  audience  of  hundreds  of  thousands  of 
players.  Through  a  controlled  lab  study,  it  was  found  that  the  game  had  impact  on  useful  visual 
perception  skills. 

The  effort  developed  a  general  online  experimentation  framework  which  was  used  to  iteratively 
refine  the  design  of  the  game.  Further,  the  effort  identified  the  key  visual  skills  and  necessary  tests 
to  apply  this  experimentation  framework  to  optimizing  skill  training  in  the  game. 

The  findings  showed  that  even  with  voluntary  play  people  improve  on  psychological  tests.  On 
fully  controlled  tests  with  pre  and  post  psychological  tests  the  effort  found  marked  improvement  on 
just  10  hours  of  play.  Specifically,  players  improved  on  visual  search  in  terms  of  time  required  for 
the  same  accuracy  of  results.  The  largest  effect  was  found  on  useful  field  of  view  cognitive  tasks 
where  people  improved  their  performance  up  to  30%  on  controlled  tests  after  10  hours  of  play. 
These  findings  point  out  that  a  targeted  adaptive  game  is  a  powerful  training  tool  capable  of 
significantly  improving  low-level  visual  cognition  tasks  with  only  10  hours  of  play. 
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Figure  3:  Visual  Search  Comparison. 
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IED 

JIEDDO 


Improvised  Explosive  Device 

Joint  Improvised  Explosive  Device  Defeat  Organization 
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